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ABSTRACT 

This is an exploratory study of sethods for the 
preparation of cosputer curricalus saterials. It deals with 
coorsevare development procedares for the PLATO IV cosputer- tased 
education systes, and draws on interviews with over 100 persons 
engaged in courseware production. The report presents a five stage 
Bodel of developsent: (1) planning, (2) production, C3) review, (H) 
validation and docusentation, and {S) isplesentation. This sodel then 
serves as a basis for the discussion of various group structures 
which range froa the independent developer to teas sesbers with 
special skills. Features of the PLATO systes did affect the group 
productivity as well as guality of lesson saterial: these influences, 
both beneficial and detrisental, appear to be isportant factors to 
consider in the organisation and sanagesent of courseware developsent 
groups. A najor objective of this study, beyond the description of 
sethods for developsent, was to set forth these characteristics that 
sight distinguish asono groups jand help in personnel selection. 
Therefore, the report includes lines of research on selecting and 
training developers. A sussary of this study and extensive 
recoBseadations for further work conclude the paper. (Author) 
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The development of Instructlorui materials', or "coursewa re^ 
ccmstltutifs a major component In the u^e of computers or other 
media for instructional purposes (Carnegie, 1972; Anastasio and 
Korgan, 1972; Hunter, et al ,, 1975), This paper reports on a 
study of procedures used by a number of groups to develop 
courseware for the FLATO^ system, Th/s conmon elements in the 
steps taken to develop usable courseware, the personnel 
considerations involved in this task, and factors which exerted 
a strong influence on-the process of courseware development are x 
described. The discussion should help to clarify the coiq>Iex 
task of courseware development and stimulate further efforts 
to reflne^the process, . 

There have been many publications which propose courseware 
development procedures (see reviews by Gagnft and Rofawer, 1969; 
Merrill, 1971; Schuta and Baker, 1971; Glaser and Reanick, 1972; 
MdCeachie, 1974), A few articles have Included descriptions of 
the operations of courseware development efforts (Teloiit 1973; 
Reed, Ertel, and Collart, 1974; Cashell, Lent, and Richardson, 
1975; Diamond, et^ al. . 1^5), while others have described the 
research or theory oa wULch lucocedures were based (Bxuner,. 1966; 
Anderson, et al ,, 1969; Gagni, 1970; Atkinson, 1972; Atkinson 
and Paulson, 1972; Merrill, 1972;- Merrill and Boutwell, 1973) « 
Further articles have dealt with general limitations of the 
courseware development process (Locatis, 1973), the political 



cbntaxts of these efforts (House, 1974; Fraley and Vargas, 1975), 
and methods for evaluating the different procedures (Felker and 
Shettel, 1975; Smith and Murray, 1975). In addition, there have 
been publlcatlong dealing with the unique aspects of Instractional 
development for computer-based systems (Bunderson, 1973; . 
Rlllelsohn, 1974; Reed, Ertel, and Collart, 1974; Broderick, 1975; 
Eahenbrenner and Lamos, 1975). However, there ha/e been no 
previous attempts to study a variety of prdcedures« Without close 
examples to follow, this study could take only a broad, 
preliminary look at Jthe spectrum of courseware development 
procedures used with the PLATO system and the factors which ' 
Influence them. Its purposes Include the following: 

1. Synthesize the procedures used by people working 
with the FIATO system through the 1974-75 
academic years Into descriptive models of the 
stages that Instructional lessons went throiigh 
as they were developed and of the personnel 
structures that were used to acccmipllsh this 
work. 

2« Raise questions about the determinants of the 
selection and productivity of these procedures* 

3. Recoomend hypotheses and methodologies for 
further research on courseware development 

* 

procedures* 



No attempt has' been made to relate this^ study to Judgments about 
the effectiveness of the courseware or the Individuals Involved. 
Therefore, final judgments about the procedures would be premature. 
Hopefully, other researchers will use the reconoendatlons of this 
paper In collecting the data necessary %o choose among the various 
courseware development procedures described. 

The data for this study were collected between June, 1974 
and March, 1975 through Intervlew^o^th 122 people who were 
involved in the development of courseware for the FLATO system. 
Wltli few guidelines concerning the specific information to be 
sought, questionnaires, on**llne data records, and dally logs were 
found to yield Incomplete descriptions of procedures, which were 
rapidly changing. ~ Interviews (see appendix) provided the 
flexibility to ask questions appropriate for' each interviewee 
and yet to cover a broad range of topics. The people to be 
interviewed were selected on the basis of their affiliations with 
certain groups of developers. All projects which were^funded by 
the agency which funded this study were included. In addition, 
seven groups supported as part of a national field trial for FLATO 
in public education* three unlversity-based courseware development 
groups an^ three support groups » organized to support development . 
efforts of othei groups » were included. See Table 1 for a listing of 
the groups » the dates of their interviews » and the number of 
people interviewed. More information about the data collection 
procedures can be found in the final section of this paper. 

10 



Insert Table 1 about here 

In order to put the discussions of procedures and their 
detemlnants Into context , the next section describes the PLATO 
system. The follovlng section deals with the stages of coutieiifare 
developoMnt and the char acteif^ sties of the coursevare which affect 
its development. The organssatjl^n and management o^ courseware 



development efforts and their Influences on the process are then 

\ i 

described. The report ]coneludes wlth^a combined sunmary and 
recommendations for furt^r research. 

THE PIATO SYSTEM 

Development of the PLATO system has been carried on 
continuously at the University of Illinois since 1959. The first 
product\f this research was PLATO I, a single tenplnal, prototype 
system using Illlae I, an early electronic computer. Subse^paent 
developmental stages, PLATO II knd PLATO I^I, were multi-user 
systems of Increasing size and sophistication* The current 
syetem, PLATO IV, became operational in 1971 and came into 
extensive use In 1972 (see Lyman, 1975 for historical 
highlights and a list of publications describing the development 
of tlie PLATO system}.. 
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features of the HATO IV System 

. The PLATO IV system hms sAveral distinct hardware features 
vbich directly and indirectly influence courseware developcaent 
>roc«dured (W6od, 1975). These include: 

1) Hasiaa display panel terminals i^ch can 
selectively display and erase standard 
alphanumeric or user^eslgned (characters and 
ll|ie drawings at any ac '.en location* 
Theae terminals can also support rear 
projection microfiche £mage selectors, 
touch-initiated input, random access audio 
devices, and a variety of other peripherals* 
The terminals can be used either by developers 
to create new software and courseware or by 
students to Interact with the c^rseware which 
has bMn developed (Stifle, 1974). 

2) A large central cooputer to which 1,000 tarolnala 
can be connected* The extended core storage or 
ECS, an integral part of this romputer, allowa 
rapid respoiise to input from users at terminals 
(Stifle, 1972). 

3) A teleeomttunications network which uses microwave 
transmission, voice-grade phone lines, and other 



devices to connect remotely located terminals 
to the central processor (Sherwood and Stifle, 
1975). 

fr-addltlon to the unique collection of hardware involved, 
another distinctive and Important component of the sysf- a 
TUIOR authoring language (Tencsir, 1975; Shezvood, 1974). 
pevelopiwnt of this language b^gan at the University of Illinois 
in 1967 (Avner and Tencsar, 1^69) and has. been refined continually 
on the basis of the experience and needs of the users (i^encsar, 
1975). Its Itey features include: 

1) Cofliputatlonal features, such as implied 

« 

multiplication and superscripts for exponents, 

■ ' w ■ 

that closely sdmlc natural algebra^plus compilation 
cf all calculations into machine code for rapf-1 
execution. ^ 

2) HuMrotts coomaiuls which enable developers to take 
adv«ntes« of the graphic capabilities of the 
play a display panel tcrodnals. 

3) Extensive ansver-^Judglng capabilities. 

4) A large selection of branching coonands to assist 
in Individual Itatlon of Instruction. ^ 

There are five discernible categories or levels of software^ 
in use on the PLATO system. These are: 
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1) System software for running PLATO prograiiia» 
including the display formatter » the TUTOR 
language compiler (or condensor)» a memory 
manager » and lnput*output support* 

2) System software for supporting and facilitating^ 
cofl^uter-based education. These Include course 
rosters or '*records^^ through which Instructors 
give access to the system^ **routers** for 
sequencing students through instructional 
segments or **lessons/* and routines for 
collection of data which summarise students* progress 
through the lessons. 

3) Systems software for assisting in the development 
of lessons. Among the most important features 
provided are: the t«Kt editor which indudea 
powerful tools for the creation of diaplays; 
AIDS an online guide to the TUTOR language 

and FLAtO system; special inter^terminal 
I communications support which allow a uaer to 
view another user^s lesson matipial^ as well ' 
as to '^talk** to the other user by displaying 
typed messages simultaneously on both screens. 
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4) Programs provided by the user for Inter-lesson 
' routing^ student data collection and analysis^ 

and special online administrative routines* 
Such support programs are generally written to 
enhance or extend the capabilities of the software 
OMntloned In 1 and 2. 

5) '"Lessons" or Instructional programs which contain 
the specific information and displays vhlch are 
presented to the student* Students using a 
condensed lesson do not see the actual TUTOR 
code which operates the displays which they see* 
Various types of Icssoni are ^escribed .n <i later 
section* ^ 



The last two categories of software are called ""courseware'* 
in this paper because both instructional ""lessons"" 9xid the 
support programs in category four had to be developed by the 
courseware development groups* The effects of developing support 
prograflis are discussed in a later section* 

Courseware has been developed in Aumerous subject areas and 

for levels oi: instruction ranging from preschool to graduate 

« 

education (Lyman» 197Sb)* These efforts can be roughly divldied 



/ \ 

I 

I 
i 

I . 

I 

i ito three time periods t as follov/p: prior to 196^-68, 1967-68 
to 1971-72 and 1971^72 to 1^75-76. \Dutlng the^l^st period, the 
PLATO system itself i^s beitig develoi)ed thorough /^rsions I, II, 
and III with primary^ fyndinji > from the U« S« Department of Defense 
Courseware develG^f>ment was nicessariiy limited to a few orgafaized 

efforts at Irfie University of Illinois. A rudimentaify language 

\ \ / 

for coarseware development, called CATO, was introduced in 196St 

/ ' 

but developers had to be quite^onversant with onher computer 
prograonLng languages. Cne teseafch project on courseware during 
the period was funded by the U. S« Office of Bducation (Easley, 
1968) • Some efforts were the work of individjlials without outside 
iupport or with snail grants from various sources. 

\ 

\ Two events in 1967-68 affected courseware , development 
efforts on PLATO. First, the Computer-based Bducation Research 
Laborajtory (CERL) was established to conl^nue research and 
developijient on thi 8yst<^m and to provide operational support for 
system users. Second, tjhe TUTOR language was introduced as a 
means ^foir developing c&u^|eware without an extensive background 
in computer use. These events led to the beginnings of the first 
four groups included in Ithis study and many other efforts. Mbst 
of these "groups" were initially Just individuals «dio became 
interested and started to work on their own. An additional 
imp'Stus to expansion was the infusion of funding from a variety 
of sources, including the National Science Foundation; the S« 
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Department of Defense: the State of Illinois; the U» S. Department 
of Healthy Education an\ Welfare; the National Institutes of 
Health; the Metropolitan Museum of Art; the Ford Foundation; the 
U. S» -Agency for International Development; the Control Data 
Corporation; Owens-Illinois , Inc.; and others. Some of these 
funds were grante!^ for specific courseware development efforts » 
but much of the money was devoted to the developoient of the 
system which was then available at no charge to anyone with the 
initiative to get Involved. 

In 1971-72 » the PLATO IV system with ft^ increase^" 
capabilities and availability^ as already describad, was 
Introduced. At approximately the same time, multi^'mllllon dollar 
projects to teat the instructional , technical » and cost 
-effectiveness of PLATO over a' period of four and a half years were 
^^unded by the National Science Foundation and by the U. 8.^ 
Qej^artment of Defense through its Advanced Research Projects Agancy 
Botli\proJects included significant cofladtmenta that courseware 
would j^e used and evaluat 1^ although specific coursewara 
development mechanisms were not prescribed. Later^ additional funda 
had to be appropriated for courseware developsMnt because there 
proved to be insufficient support available for these efforts. 
Mont of the groups included in this study received support from 
these two projects and hence may be a biased sample. An 
effort was madej therefore, to^ include three other groups even 
though the wide variety of small efforts could not be included. 

s 

« 



At the same time that CERL received these large grants » the 
Educational Testing Service received separate contracts to conduct 
an independent evaluation of the NSF project and to facilitate 

instructional research related to the ARPA pripject (Anastasio» 

\ 

1972; Alderman and Mahler, 1973). The later contract Included a 
commitment to study the courseware^ development pro^dures used 
vlth the PLATO system* This paper is the report of \that study; 
reports on other aspects of these projects will be issued 
separately* \ ' 



COURSEWARE 



Stages of courseware development 

In order to provide a focus for the following discussions of 
the detfsrminants 6f courseware development procedures, this * 
section begins with a generalised description of the procedures 
under investigation* This description shares tsany characteristics 
with the numerous general instructional development systras which 
have been propose4 or described (for exaiqple, Gagnii, 1962; Glaser, 
1966; Smith, 1966^ Biggs, 1970; Schutz, 1970; Gerloch and Ely, 
1971;. Johnson and Johnson, 1971; Kemp, 1971; Popham, 1971; Baker, 
1973; Eisele, 1973; Pents, 1973; Wallen, 1973; Gagni and Biggs, 
1974; Gow and Yeager, 19^5; Kozma et_al., 1975; McManus, 1975). 
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There Is also correspondence with a proposedX^omat for reporting 
on PLATO lesson development, which Includes th^ stages of general 
planiilng, preliminary design, final design, form^i^ve evaluation, 

in^lementation, summatlve evaluation, and maintenant^e (for a 

\ 

description of these stages » see Avner, 1973). The d^cription 
given here had to be made more flexible and less prescriptive 
than previous descriptions because of the vide variety of 
approaches used by PLATO developers. Some groups have adopted 
rather elaborate and structured procedures, vhlle others have 
proceeded on an ad hoc basis. Figure 1 is intended as an abstra<it 
suDomury atid not as a description of any particular group or 
approach. 



^1 'insert Figure 1 about here 



V 

Planning . It 1^ possible to start with any of the three 
activities which make up the planning stage (see Figure 1) • Zn 
many cases, the contend or behaviorkl objectives were defined^ an 
appropriate instructional strategy was selected, and then a basic 
program structure was created or adapted from an existing lesson. 
In other cases, the developer began because ojE well<»fomulated 
pedagogical reasons with a commitment to a particular instructional 
strategy, such as drill and practice or simulations, then 
identified appropriate content objectives, and finally specified 
the ftrogrsm structure. In rare cases, a developer chanced upon 
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an Interesting program, usually a game» and then developed 
objectives and selected content from a different subject area to ^ 
fit that fonaat. 

The depth and breadth of material being planned varied across 
groups and time. Fmir of the thirteen productlon*orlented groins 
started with the Idea of developing curricula for single courses or 
multl^year educational programs. Five groups and some mesibers of 
another group started with individual lease, i and then later planned 
entire curricula. The other oiembers of the split group «nd^llbothftr . 
group planned ii^lvidual lessons which were never expandiid into 
entire curticula. Two other groups decided to develop lessons iihich 
fit into an existing cUrriculttm, but once the topics had been 
4atermined« each lesson was planned independently.^ ' 

The formality of the planning process ranged from the personal 
thoughts of an individual to written proposals which were reviewed 
by a coopsdttee* Likewise^ refinements to plans ware based on personal 
comcmnlications between developers in four of the thirteen production* 
oriented groups, while the other nine grojps had mora formal amchantima 
such as revised proposals or group meetings, ^^^^^was found that in 
addition to revising the plans for individual lessons, twelve of the 
thirtim groups msde changes in their overall goals. Such changes 
ineluiled new target audiences, reductions and expansions In the breadth 
of csverpge, reductions and expansions in the depth of coverage, 
reductions and expansions in the number of Institutions to be served, 
and changes from a research orientation to a production orientation. 

\ 



14 

Production > At seen- In Figure 1^ production is the center 
of the entire coursevere development process. Ir entails the 
fshrlcatlon of the design specifications of the planolng stage 
and the modifications, suggested by the review and validation 
ateges. It Is complete; only when the lesson is raady for 
<»perational use by students. Production was largely carried out 
by individuals who were sufficiently proficient: with the TUTOR 
language to program the lessons under developoenty although 
non^programmers fre<iuently furnished suggestions and ideas'^ 

Mnce production involves all coding and debugging of 
lessons » it Wes generally the mosl^ time-conswdng stage of the 
process «^ In general, the amount of production work vas reported . 
to be Inversely related to the oiount ol detail itt^tha design 
apeciflce^ions and directly related to the number of revisions 
cauaed by ctltlcisms» the discovery of omissions » and the creation 
of new ide^es from other stages. Although there ia 90 conclttsive 
evidence, i^ appears thet production time vas atso affected by 
the complexity of the lessons, the u^e of intra* and inter^lesaon 
\ connactionsp the use oi^ cos^licated graphic displays, and the 
^^ed for special data collection rputlnes. 

Almost all production work by the vast majority of the 
developers vas done vhila using a PLATO terminal. While a very 
few^fWj^e vent to the extreme of writing out their TUTOR code 
beforfhand^NMSt people N^ad only a gnfral notion of thm code they 
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vera about to produce when they sat down at a terminal « Some 
people developed flowcharts » particularly for coiq)lex support 



routluesy before they began programming on-line « Most developers 
used printouts of their existing code when making revisions » \ 
partly because the limited size of the FIATO screen does not 
allow all parts of the program^ which may have many interactions, 
to be viewed simultaneously* One reason for the great^afQi9unt of 
on-line work was the ready access of the system beginning in 
1972-73 « Another reason is the availability of many on-line aids 
and the capability of testing out new sections o£ code immediately* 
One developer contended that since interaction witb computer-based 
courseware is very deiirable and beneficial, it la only natural 
that experienced developers would use a great deal of on-line 
interaction in their own; work habits* 

Review. Not only were a variety of pebple asked to critique 
lessons^ but their opinions on the quality o£ the Instructional 
strategy^ the efficiency of the. coding, and the accuracy oit the 
subject matter content were all the targets* o£ reviews (see 
Table 2). In some groupa this process'^as quite formalized and 
was st^ervised by someone other than the ptlmar> ueveloperr , 
(Prancla» GoldateiUt and iSweeney^ 1975; Essex^ 1975); ^ 
In othet groupa each author informally solicited comments 

according to personal preference. While personal, face-to-face 
review by a peer was the most common form of review, some groupa 
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foimd It beneflclaf to have critiques ^Iven in written form 
because of the f;reat p^rspnal* attachment that some authors had to 
their lessons (Francis, Goldstein, and Sweeney, 1975; Essex, 1975) » 
In addition to critiques peers and experts, the review stage 
incluaed Inijtial student . testing , which was often done with 
fragsMnts of lessons as well as complete lessons. While some 
groups collected aiui analyased on-line data and wrttten comments 
from these first students; most information from initial -student 
testing was c<iilecf;ed by watching the first few sttlde..ts and \ 
observing problems as they occurred. This procedure also allowed 
^he developer to furnish assistance if the student could not 
proceed. 

As Itible 2 Indicates, other developers, who were usually from, 
tlie same group as the primsry developer^ jof a lesson, were the liost ' 
common reviewers. Most groups alsd^ditf some type of student 
'testing before making a lesson generally available or using it as 
part of the regular instructional program^ The feiw external 
ttcperks wlip were used as reviewers usually looked at only the 
overall goals of the project with specific lessons as Mwiples* 
The eontent and instructional design received^ about the same, 
amount of review and were often tjrevlewed sioultaneoasly. ^ The. 
ac^al TUTOR code received less review, as would be expected, 
and iuch reviews usually focused on Identifying potential 

•ends** or other prograoning %ugs.** 
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Insert Table 2 about here 

_ _ _ < 

Validation and docimentatlon. Although soma PIATO lessons 
have been validated and documentedj^ many have not yet reached this , 
stage of develoiMsent or were lntede4 for private use. Proper 
yalldatlon and doctiiMntatlott can be very useful In encouraging 
the widespread usage needed to j>i8tlfy the substantial «cpendltures 
of time and resources used In the previous stages of coursware 
development • Avner (1973) encouraged that .documentation and 
validation be buUt Into the entire process of lesson develo|mient 
and not sliqply added on the end* Re advocated Mn Iterative 
approach which creates IncreaslAg specification of the target 
audience^ obJectlyes» Instructional strategies, and validating 
evidence as the lesson' Itself Is developed* Avner (personal commimi 
cation) suggested five types of validating data which can he easily 
collected by the system: student time spent, degree of luiteractiont 
lesson difficulty, anticipation of student needs, and relation to 
external measure of achievement, attitude, or behavior • 

Host of the documentation of PLATO courseware that exists 
thus far focuses mainly on descriptions of the content and 
Instructional design (see Lyman, 1975 for a list of publications 
about PLATO). Statistics on usage, reactions from students and 
instructors, and records of student response data collected by 
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the 8y8t^ are also included (for example, Alpert atid Jordan, 
1$76). Published confer i sons of PIATO courseware and other forma 
of instruction have been written by McKeown and Lenehen (1974), 
McXeown (1974), Bennett (1^/5), Dare et al , (1975), and Konta»ellt 
(1975). In Addition, there are reportedly other internal or 
unfinished studies of the effectiveness of PLATO courseware. 

Implementation , The three tasks of tlie implementation^stage 
a^e quite distinct (see Figure 1) . Maintenance of lessona 

> includes the monitoring and analysis o^ data being continuously 
collected, making minor programming changes in response to system 
cKangeii, and adapting ^materials to fit the needs of new users, for 
exa^le, changing the sequence of certain lessons or the type of 
mathematical notation used. Tl^ groups involved In the NSF 
projtet were carrying out such tasks, but most other groups were 
not yet providing such services to other users* Updating 
materials implies substantive changes which should bt undertaken 
only by a qualified professional, presumably with the permission 
or cooperation of the original developers* Such changns could 
teflect new knowledge in the subject area, system capabilities 
which were either unavailable or simply unused, or improve^ 
Instructional strategies* Ifone of the groups were yet updating 
leaaons at the time of this study. Groupa were only disseminating 

^ their coursaware^upon request or part «f the HSF*sponsored field 
taat* Such activities included diemonatrations, training sesslofta* 



w 

consulting on proper usage, and other efforts to encourage use by 
faculty at the participating schools. In addition, there .vera 
people vho vprked At the schools and were responsible for keeping 
the PLATO opet'ation functioning and making the materials 
acceasible to students. (See Mahler, 1976 for m more thorough 
discussion of ifj^lefsentationO 

Since the interviews for this study were conducted, 
continued con^ct with CERL has led to the observation of ' 
increased implementation activities^' In all cases, origiital 
developers were Inltlelly involved in the malt^enance end 
dissettinetlon acilvltiesv More recently, new personnel "hcv^ be^ 
•dded'^to eseift with such efforts and|(.lln a feW easC^i, have 
etsueed prinary responsibility for impl ^ntetion when the 
original developer^ hfye moved on to new projects^ IRiile 
iapleoMnUtion hes fvuMtioned quite seperetely frosi the prstvious 
stages in nany ways, it has been «iasier;^when it was, planned ttm 
the beginning rather than added tp. the devalopment of c^oura^hmria . 
designed fol use only by the developers. J 



Controlling progress ^ It became neceasairjf in wm tii* the 
thirteen productiofl*oriehtiid groups for a sieebiiuitipi tli> Ibe 



eatMlished for J^eping courseware oioving through dui various ^ 



ata|is## The MjoJj bottleneck was usually 'at the productiwk kt;age 
becllituae there were: many loops lAich fed back to further refinement 
and becafise some d^elopets became so engrossed in lei«9ni that^ 
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they were never satisfied with their qmlity. Another bottleneck 
was caused by having to schedule ''outsiders'* for reviewing or 
student tasting. Fiiyilly, documentation,, maintenance, and 
dissemination were not particularly appealing activities for ma^ 
courseware developers and were sometimes postponed indefinitely 
in favor of developing new lessons. 

Two mechanisms were used to insure the orderly advancement 
of lessons through the various stages. The first approach was to, 
have lesson developers fill out a form as they coii7.1eted each ( 
step. Thts reporting Jcept them continually aware of the remaining 
steps and the current status of each lesson. The second approach - 
was to designate a particular persoip/ either the group leader or 
the group evaluator, to monitor the progress of each lesson a:id 
intervene when a problem arose. Such persons operated either 
formally with various forms and scjheduled meetings or inforaally 
through persuasion and personal contact. Each of these procedures 
added significantly to the amount of administration needed by the 
group* 

3uch proce lures were needed mostly by production-oriented 
groups with large and sometimes coiqilex organizations. They were 
usually used only when ext.irnally lii4)osed deadlines from the 
sponsoring organisation needed to be met. Another contributirg 
factor was that very few developers had ^uch prrvlous experience 
in curriculum deve^w^ tent or computer applications. 
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Because they were learning how to use a new medium and how to 
develop courseware, it was difficult to estimate the time required 
for completion of each stage. Since the PLATO system was adliilng 
new hardware and software capabilities during this same period. 
It would have been Impossible for people to be thoroughly familiar 
with this new medium. In fact, some of the courseware development 
groups had to be deeply involved in the hardware and software 
developments which were needed for their courseware. These topics 
are discussed more thoroughly in a later section on the Influences 
of the PLATO syatem« 

Comparison to other education S & D efforts . The primary 

empiiasis. of the.PLATd project has been to develop a flexible,^ 

*' ' ' ' '^?"'*. • 

education^oriented fjji^lrdware/software system rather than to develop 

curricular materia^di land a medium to deliver them. Th^ leaders 

of this effprt hkve ^een .engineers and physical scientists, as 

well aft other educators. Easley (1968) suggested, as an Ideal^ 

a .team->approach to cours^are development similar to the 

general instructional development syatm previously discussed. 

As more capabilities were added, dissatln^factlon with a **8ytit«ms ' 

approach** grew (Avner, 1975). In an effort to maintain maximum 

flexibilicy and to explore all possible uses of this relatively 

new medium, developers were encouraged to use their creativity 

Individually. By the early days of PLATO IV In 1971*73, the idea 

had developed that all - ^ 
a 
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college«*l«vel Instructors could write lessons for use^ In their 
mm ccmrses, presumably eliminating the need for validation, 
documentation, or dissemination of courseware. This belief Is 
■ still espoused in an official PLATO publication C*od, 1»75)* 

The great Influx of fundli^ In the early 1970*s caused two 
problems for this approach First, additional deadlines and 
Independent evaluations were required by the funding agencies* 
Therefore, wasted efforts caused by trying out new Ideas were 
less tolerable* Secondly, many more people were enticed into 
trying to develop courseware by full-time positions, release 
from teaching duties, the potential for royalties or Job 
advancement, the glamour of a nationally recognised project, and 
t4.8 personal encouragement of the project staff* Without the 
previous self-selection hurdle of strong personal dedication in 
order to withstand the uncertainties and problems of working 
with a developiiig system, many of these people proved to be 
unproductive (GJerde, 1973; House, 1974). For the past few 
years, there has been a shift back toward the more structured, 
team-oriented .groups. Personnel and group structures will be 
discussed later in this paper; the point here is that the final 
stages of courseware development are once again receiving greater 
emphasis* It should be clear that there are advocates of both 
structured and unstructur'tedJspproaches to PLATO courseware 
development* The ultimate value of each approach cannot yet be 
judged*. 
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Since most of the literature on instructional developaent is 
prescriptive, that is, telling what should be done, it is not 
surprising that most of the models proposed are quite formal, 
elaborate, and rigid. The few papers which are descriptive of 
;rhat has actually happened (Cashell, Lent, & Richardson, 1975) tend 
toward more flexible approaches. When the details of some rather 
structured models are examined (Reed, Ertel, & Collart, 1974) or 
when the people who actually produce the instructional materials 
are interviewed, it becomes apparent that many ad hoc decisions 
must be made and revised In most cases, and an iterative approach 
which includes modification of objectives and strategies is the 
rule rather than the exception. While PLATO 's hirtory of courseware 
development has. Varying emphases and many variations in procedures 
^ still exist, the overall effect does not seem to be^ very different 

froa other educational R&D efforts. 
Types of lessons . 

Several different classification, schemes for describing 
<Mwputer-based courseware have been proposed CZitm, 1967; Grubb, 
1972; Cody, 1973; Milner and Wildberger,4974; Wang, 1976). They 
contain between five and thirteen categories, which are usually 
placed on a continuum jf "increasing student control" (Milner and 
Wildberger, 1974) or "a progressive shift in the locus of control" 
froM the designer to the student (Grubb, 1972). 
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Vang*8 Index to Computer Based Learning (1976) contains 
descriptions of 336 PLATO lessons or groups of lessdns. Each 
entry Is supposed to Include the Instructions! strategy used, but 
unfortunately 225 o£ the PLATO lesson descriptions slaply say 
that they used a "mixed" Instructional strategy. The remainder 
used the follovlng descriptors: drill and practice (56) » 
tutorial (37) » simulation (12) » computer-managed instruction (11) » 
problem solving (7), inquiry (6), gaming (6), diagnosis and 
prescription (1), socratic dialogue (1), testing (1)» and 
Intrinsic (1). Unfortunately, many of these terms are used by 
various groups in quite different fashionst and Wang does not provide 
explicit definitions. The total nundber of descriptors comes to more 
than 336 because several entries used more than one descriptor. 
With the large number of entries using the non-descriptive 
"mixed" descriptor and no information about hov the 336 entries 
were selected » no^condusions about the Instructional strategies 
of the 4000 hours of PLATO Instruction claimed by Lyman (1975) 
and Wood (1975) can be reached. 

There are presently no complete data on the relationship 
between lesson types and work-hours, personnel needs » or sny 
other courseware development procedures, tn a preliminary study » 
Avnar (pexsonal communication) found that first^^ear developers 
on the PLATO III system took more time to develop tutorial 
lessons than drill lessons, possibly because the former simply 
use more code. 

\ 

\ 
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Scop€ of the courseware > ^ 

The beet way to organize a courseware developnent group » as 
veil as the amount of work, is greatly Influeiieed by the depth 
and breadth of the courseware to be produced. The scope needs to 
be measured along at least three dimensions. The first and oiest 
obvious Is whether the materials will be used In an entire course, 
for A single unit within a course, throughout the entire 
curriculum of a depart^nt or school, or to some other degree of 
breadth. Secondly, there needs to be a decision regarding the 
depth or degree to which the coDq>uter«based materials are expected 
to teach the content. They may be Intended for use as supplementary 
omterials for those who want enrichment outside of clasSt as 
temedial work for those who cannot keep up with the regular class, 
as drill or practice on concepts taught In other ways, as the ^ 
primary source of instruction for selected concepts or activities 

(such as siqulated labs), or as the sole source of instruction 

r 

for students at remote locations. The terms ''adjunctive vs. 
mainline'' (Bunderson, 1973) have been used in reference to this 
dlmensicn, but it is at least a continuum if not a oatltl-dimenslonal 
ieoneept# The third dimension is highly conf oundfed^ by the first two 
and refers to the degree to which the individual pArts of the 
courseware are expected to be related and coordinated. At one 
extreme, if supplementary treatment of single concepts is planned, 
coordination could be extensive but will probably be nanimal* If, 

?2 
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hovttver, primary instructioij throughout an entire curriculum la 
planned t the individujA lessons should be closely related to one 
another but could exist as a ^^smorgasbord" of possibilities to be 
selected4>y the individual instructor or student. 

As the scope of the courseware increased, not only did the 
amount of coordination of the work increase, but the^e was also 
a greater need for planning and review. 

Content areas. 



It is not clear at this^time that any broad content areas are 
not suitable for computer-based instruction; rather. It appears 
that certain types of instruction within each area ai^e better 
suited for this mediiim than are others. Since PLATO lessons have 
been developed in more than fifty content areas, ranging from the 
kindergarten level to graduate courses » no area of education can 
or ahould be excluded ct this tine. Merrill (1975) has auggeated 
some criteria for deciding if a particular Instructional sequence 
should be programmed on PLATO, but there is certainly room for 
apprdprlate' topics in all major content areas and age levels and 
for expansion of these criteria as new features are created. One 
coneluaion that can be stated is that courseware production la 
eaaiest ^en the objectives can be clearly atated, and thla 
process may be easier in some areas than It is In others. However^ 
eaae of development must be weighed against potential benefits and 
the need to explore new areas. 
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- 9RGi^lIZATI0N^MID HANAGEHENT 
ChMif^in^ ettphmtca for gyoup activities > \ 

All of the groups 1)& this study shifted the emphasis of th^eir 

vork activities over thii course of time, ^he first emphasis was\ 

\ 

characterised by explar^tlon of PLATO *s potentials. Some plaimad \ 
for this type of activity while others were forced Into It becausm of 
false starts and chamtek In goals. While most Interest vaa on 
the^nnl^ue featur|^ of the system, there was also Interest In the 
appro/rlaVe\^^ strategies. This first edq>hasls lasted 

from a oionth to more than a year. The "second emphasis was on 

^veloplng Instructional lessons, either to fit into a grand 

/ f 

/scheme or as Independent modules determined by the interests ot 
the various individuals. These lessons reflected the possibilities 
explored earlier and used material devcgp6i^ed during that time, but 
the emphasis shifted to producing, rcvieiring, and revising lessons 
until they vere useable* The third emphasis vss an attei^t 'to fit 
all o£ the lessons together into the on«going curricula df ^one or 
s»re^rei^lar dassrooott and to validate their use. This effort 
often liiacessitated modifications, the development of a routing 
system, and continual liaison with the classroom instructors « 
particularly if such activities vere not; previously included. 
The fourth end final emphasis is characterized by the maintenance 
of the courseware and attempts to spread its use to other 
dassirooms. At the time the interviews for this study were 
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conducttfly no groups had reachad this, stsady state. ' Since that 
tlat, elnost all grou|^s have moved into this i^lensntatlon stage» 
and Mst have found thajt the developers have left and that this 
WDxfc Wist b« carried out by staff nenbers hired specifically for 

Model d of group structure # 

Bach of the 16 groups had Ita own unique organisation and 
hi^orya To describe e^h of these vould be tedious and 
confu^inga Therefore, this discussion begins vitfi four abstract 
■odels .of group structure , vhlch were based on an analysis of the 

\6 groups^ and then con^aref the groups vlth these models* 

\^ 

the independent developer » Sonellmes one person took 
responsibility for the developoent of instructional lessoda« Such 
a person often sought advice but made, the-f Inal decieiona and did 
most of the vorka They often shared ideas and crleiquea vlth 
other developers, but there was no direct coordination, organised 
group effort, or formal relationship in most cases # Independent 
devel^rs needed ei^ertise In a content area^ inatroctional 
design, and TUTOR progranBlng and have been c^alled PLATO ^authors* 
It was hoped at one time that maiqr college fiaeolty members nould 
becolM independent developers (House, 1974; Wood, 1975). 

The c^ ^ taaaueship a Sometlmsis several people of essentially, 
status worked together vl.th a commitment to cooperation 
deeislon-maklnga In most cases, specialisation occurred 
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or wms plaxined. In one group* one person took responsibility for 
adainlstratlve duties, another became an authority on intricate 
displays, and another specialized in observing student bdia:vior 
and evaluating. However, all members retained responsibility 
and Interest in the total effort, and no clearly defined hierarchy 
emerged. ' 

The lesson designer vith programming assistants > Subject 
Mtter experts, either faculty members or full-time developers, 
sometimes specified the content objectives and Instructional 

strategy and then let a progr^modng assistant design the actual 

• - ■* 

lesson. These programming assistants. Who were often students, 

worhed vith varying degrees of autonomy depending upon their 

own cSntent and programming expertise and the desired involvement 

of the lesson designer « Contact between the two ranged from 

Interactions on a dally basis to only formal reviews of completed 

vork at intervals of a month or more. Some lesson designers 

vould begin the actual programming and then let an msslstant 

finish it, while others, who were not knowledgeablm about the 

FIATO system or compute^based education In general t bad to rely 

wch more heavily on thei^ programming assistants. 

The support staff * Several groups added specialists In 

TUTOR programming, instructional design ai^ evaluation^ or 

audio-^lstial production. The TOTOR experts consulted with lesson 

designers and programming assistants, worked on partiisularl^r 

3G > — — \ 
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cofl^plex routines 9 especially support routines i^ich as '^routers'' 
•ad "drivers 9 and trained new group nembers to usa the TTJIOR 
language, thm Instructional Resign and evaluation specialists 
consulted with lesson designers and were often reaponslble for 
reviews and .validation. The audlo^vlsual produ^ion specialists 
were used irtien there was a majdr cosnitaient to the use of 
■dc^roflche or audio messages^. 

Cospartscm of podels and groups ; The correspondence between 
the Mdels Just presented and the groups irfilch were interviewed 
for this study is shown in Table 3. The groups are. listed 
aeiu>rdlng to the dates when they first began work on FLAIO. The 
various phases were used to Indicate dlffer^ent tines at lAlch 
groups changed their nodes of operation. The length of each 
phase varied with each group. 



Insert Table 3 about here >y 



Sooie of the "groups" -were not actually organized structures » 
particularly in their first phases. All of the non->adlltary 
groups started as several separate efforts » each often ^xfaibitli^ 
the characteristics of different models. Sosetines these efforts/ 
were located in different departments or even different institutotons 
4Ad had very limited contact with each othee^ Formal group 
structures and coordination activities came only when outside 
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funding pemlticeil release from teaching duties or full-time staff 
members. The i|dlltary sites all began With such funding. 

Three of |he groups were studied because, they provided 
siqpport for otfer coursemre development efforts. Although the 
people In each ^ of these groups did develop Instructional lessTons, 
often as training materials or on-line aids for iither' develop€rs» 
their primary Kklsslons were not to develop lesson^ for studrat 
use. When they did produce lessons, either individual members 
vorked tlosely with another group or the groi^ functioned as a 
colleagueshlp to produce support routines and training materials. 
Because of this unique status » they are listed in Table 3 as 
pure forms of the support staff and are not Included in most of 
the following discussions, which focus on the thirteen 
''production-orlented** groups charged with developing lessons for 
student uee. 

All but one of the productioh^orieiited groups exhibited the 
chwdcteristics of two or more models. The lone ei^eption made 
such heavy use of the Military Training Centers group for support 
that it cannot completely be considered as a pure exaiq>le of a 
cblleagueahip. All of the nonrmllitary groups without support 
staffs, and many with their own support staffs, made use of the 
PIATO Services Organizatlon^nd the PLATO Education^ Evaluatf -^n 
and Research group. 



It should be noted that four of the first five groups end 
fpur of the last five groups have only one phase, vhlle the 
groups In the middle of the table have more cosqileoc patterns. 
There appear to be several reasons for these chancteriatica« 
The four early groups with only one phase vera all baaed upon 
substantial Involvement by University of Illinois faculty msnftera 
who demonstrated great interest and conmitment early in the 
history of FIATO. They seemed to have known what they wanted to 
accomplish and how they wanted to work nbre clearly than 
developers hired for special projects. Also because of tl^s baae 
and the early starts, tl^se groups were under less scrutliqr than 
th« raulnlng groupi, which were all funded by MSF or ARFA. 
Therefore, some of their early phases or more subtle changea may 
have been miased. On the other hand, the latest one-phaae groupa 
vera all either support groups or military groves and hava 
relatively short hlstorleiB. Not only m^TSnqrlif^a banef ited 
from the mperiences of earlier groups, but their non^^cademlc 
baaes had special consequences. Thus, we cannot be sure whether; 
the lack of any independent developers in the laat aeven groupa ! 
la tlM reault of disenchantment wi^h that mode of operation or 
la because military sites mitigate agalnat such individualism. 

Finally, most of the groups with convoluted histories ftre 
part of the National Science Foundation project and thua wer^ 
gffMtly Influenced by a second Infusion of funds and accompanying 
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nadj available for general uae. Without any coordination, there 
Vttre examples of several Independent developers who worked on ^ 
lessons covering the sane topics, and there was sooe tendency for 
Independently leveloped lessons not to be used by dther ^ 
instructors. On the positive side, independent developers 
required little administrative overhead, and the more successful 
ones often worked for Intrinsic rewards or professional 
recognition, rather than for full«time salaries. Also, some of 
these independent developers produced very creative and apparently 
effective lessons. 

The colleagueships, while not usually reqtiiring a great deal 
of formal administration or direction, did need a subsLantlal 
amount of effort aime^at fostering cooperation and group spirit, 
Findi^ and keeping qualified people was also a problem. Since 
they were all of equal status, there was little chance of reducing 
costs by paying lower salaries for routine work. Because of the 
specialisation that occurred In all colleagueshipS|» there was 
less demand on each group mtfiber to provide expertise: in all 

facets of courseware development. Lessons using sophisticated 

\ 

programming were possible without causing an undue buiiden on any 
individual to learn an unreasonable number o^^new skillc. The 
sharing of planr and mutual reviews of lessons eliminated most 
of the factual errors^ inferior instructional strategies, and 

/ 

/ 
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poor prograiming. ^he popularity of this group structure is 
evidsnt in its being part of eleven of the thirteen 
production-oriented groups » 

Lesson dasigners vtoe mostly faculty metayers or. full-time 
content exper' 3 with many responsibilities. Adding progranmlng 
assistants see&ed to make better use of these higher-paid, busy 
content experts. Several faculty members, who would not otherwlaa 
have done so, became involved with PIATO : ^ use hey did not hava 
to do thtair own programming. The success 04. these arrangements 
depanded upon the programming abilities of the assistants^ the 
amount of lavolvcment of the lesson designe.;8, and their ability 
to Communicate vlth each other. Success was V^ry limited vheil 
the programming assistant was unfamiliar with the content area 
and the lesson designer was unfamiliar with the FL/>TO system and 
cooiputer-basad education, in general. Not only was coipimlcaMcii 
diff£cult in these circumstances, hht creative interaction 
between the content specifications and the instructional design 
specifications was almost iiiq>ossible. 

Suppov; staffs were either incorporated successfully with 
other group structures or were used extensively by other groups « 
TUTOR escperts were the most fomnon type of sujpport t.taff members. 
Bvaluation specialiats, who usually had advanced degrees, were 
often primarily responsible for reviews and validation efforts. 
Some of the audiO">visual production specialists had advanced » 
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degrees and were responsible not only for the production of 
•114es or audio discs but also for their design. Instructional 
design consultants were the least used type of support ftaff . It 
Is unknown whether the additional expense of an IntsJ^Oil support 
staff can be Justified by greater productivity by the primary 
developers or greater cjuallty In the courseware. Support staff* 
tended to be added as groups became larger and more complex, and 
Lit seems likely that sophisticated prngrams, validation, and 
^e^lpheral devices would njt have been used without the help of 
these specialists. 
Influences of the PLATO System 

A number of factors which Influenced the organization and 
management of these groups were determined by the use of the 
KATO system aad were not under direct control of the individual 
group'i. These factors often had differential effects on groups 
but operated mostly as a general context in which group decision* 

could be made. 

Development of the PLATO System . The CERL policy of 
continuing development of the system and its language (Avner, 
1975; Steinberg, 1975; Stifle, 1975; Tenciar, 1975) was 
advantageous in some ways but disadvantageous in others. It led 
to a system which is very flexible and well suited to the needs 
of a broad spectrum of educational applications. Author input 
had a significant impact upon the develqpment of the TUTOR 
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language » as authors could request and usually receive new 
language featuers to facilitate their work. Of the experienced 
TUTOR programers who were interviewed, none indicated ever being 
limited by the language. This continuing development has led to 
efforts in many areas and with many instructional techniques 
that could not have been imagined without ample eiqierience with 
the medium. Hovever, this expansion and refinement also created 
demands on programmers be .ause it was coupled with instability* 
If someone did not use the system for a few months or weeks t " any 
new or revised features often had to be learned before productive 
efforts could resume. Some people h^ve spent considerable time 
in efforts only to find tha.t a new system feature made the 
problem trivial. There have, also been soira people who found 
PLATO programming to be so confusing, demanding, and unstable that 
they decided that it was not wor^rh the efforts required of them» 
As more PLATO systems are put into operation, cmtlnued development 
of the authoring language and system software will probably be 
restricted to one experimental system with the operationally oriented 
systems receiving updates at less frequent intervals. 

Hardware and software limitations . Although. the power of 
the PLATO hardware ami software have created many opportonitias 
for innovative educational use, several problems arose in the 
early period of courseware development which adversely affected 
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that process. These problems were largely created by an 
unanticipated competition for PLATO system resources. Initially » 
■any groups, especially those remote from CERL» had difficulty 
obtaining tenalnals, often due to. late delivery. Aa veimlnala 
became more widely available, authors found development hampnired 
bacause of shortages of "lesson spaces"~allotDient8 of disk 
storage. These shortage3 have generally been alleviated by 
acquisition of hardware. Eventually, critical difficulties were 
experienced with the allocation of Extended Core Storage (ECS). 
As more authors gained access to PLATO and as student usage 
increased sharply^ it became dear that there was insufficient 
ECS to suport the resulting number of users. Although additional 
ECS was obtained, it became obvious that this resource bad to be 
carefully managed. * 

Further restrictions were placed on authors because of 
competition for the "condenser," a program which translates the 
author's program into a format usfed by the PLATO computer. There 
was not enough computing power to allow authors to condense at 
will (sometimes several times a minute while debugging) » so sharp 
curtailment of "condensing" was enforced. 

CP 

In all caaes, students were given higher priority in access " 
to these features than authors, so that much on-line work during 
the day 'fight hours, when students were using the system^ became 
difficult and sometimes impossible. Developers were frequently 
forced to adapt to a nighttime work scb'^dule. 
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Development of some aspects of the system legged behind 
others; In particular » peripheral devices such as tfie'touch 
panels slide selector r.nd audio device had not reached a stage of 
development coiq>arable with other hardware on the system. The 
touch panels, though reliable , were produced slowly at fltst and 
were consequently not readily available. On the other hand, 
slide selectors were widely available, but microfiche for the 
slide selectors were difficult to produce. The production 
process was rather \t.^ (two weeks at minimum), and the quality 
was not fonsl stent. The prototype audio device was both } 
unreliable and In short supply. Audio recordings were difficult 
to produce, and the end result was frequently undesirable. In 

^operation, the device was often balky. 

As a consequence of. these problems, those groups which 
counted heavily on the operation of these devices In their 

^coursewar^ development were required to devote time and resources 
to maintenance and repair of equipment and feedback to design Engineers. 
In some cases, it was necessary to redesign and reprogram lesabns to mlnlalza 
dependence ^n the peripheral c'^lces. Subsequent improvements to ^ 
the hardware and productlon^processesf have reduced the early 
problems with touch, slide, and audio so that future developers 
should not encounter the same difficulties. 
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Proxlittlty to CERL, Having « group based at CERL had several 
advantages and a few disadvantages/ First of all, users in 

Chaapalgn-Urbana needed to be far less concerned vlth routine 

'» *»• 

hardware nalnteflpnce and reliability. Many experienced, 
technicians were close at hand» and there iiere fever probleias 
caused hf telephone tr nsmlsslons and delivery of equipsMnt, 
especially during the early dayd of the project.^ Secondly, 
esifperlenced PLATO progranners and lesson designers vere r^ily 
s^vallable for face- to*f ace consultation, while renote users had 
to depend Bu^re heavily on oilrline consultants located at CERL. 
Thirdly, this pool of experienced people located at CERL could 
often be drawn upon for staffing a new project. ' 

Sources of advice . Most of the advice which was sought 
concerned how to program a particular sequence. Although some 
people first tried to get the information they needed from the 
on -Hne documentation called AIDS, most vent to a fellow prograsRier 
or the TUTOR expert in their group. As the Ains materials have 
been Improved, they have been used more»»even as the initial 
source of information. When & local resource person was not available 
could not help, they contacted a member of the PLATO Services 
Organisation or some other TUTOR authority at CERL either 
personally or through the on-line consulting capability, which 
allows both the consultant and the client to look at the 
appropriate part of the lebson while typing messages to each 
other across the bottom ot the screen « 



8tne« noat dcvtloper; had already decided on the content atid 
Instructional strattgy of the lesson nhen they ber^en production, tliey 
usually did not seek advice in these areas. Advice on these aspects 
soMtines came during training ^r the planning stage of courseware 
developsMnt but was onlx actively sought during the revieir stage. 

Use of support routines . The parts of the coursetrare which do 
not contain tiie content-oriented instructional naterlals but which 
serve In the developaent or presentation of such materials are 
referred to as support routines. These routines nay be integrated 
into the InstructloneL lessons or may exist separately, using pr 
being used by the instructional lessons as needed « At least three 
types of support routines are in use. '^Drivers" are routines which 
set up the basic format of a type of instruction and into which 
different content can be placed. For exa«f>le, some ^^drlvers** 
provide the basic structure for multiple-choice questions, 
including placement on the screen, random ordering of altematlym 
responses I corrective feedback , and storage of responses. It ifi 
then possible for someone with little programming experience to put 
in i{uestions with appropriate response alternatives and designation 
of the correct answs^s. A large variety of these **drivera" have been 
vritten and made available for general use by theK developers. A 

" -V 

second type of support routine is for the collection and manipulation 
of atudent data, f Some data can be collected automatically by 
aysten-provided routines. This type of data is intentionally rather 
general, with more focused data collection being left to the individual 
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d«v«loper according f his <nm specific needs. In addition to 
tlitt' col lection data, it was also necessary to provide programs 
for data sunnary aiul analysis (Tatsuoka and Siegel, 197^. The 

c 

third type of support routine Is the ''router** vhich routes or 
transfers studenta to the Inacructlonal isaterlal lAlch th^ should 
see next. A systea^provlded router has been developed, hut iom 
groups used elaborate routes of their own design. A heavy 
conltiiient to the ub^^ support routines had several Implications 
f6r courseware production. First of all, they usually retired 
the skills of an experienced fulltlne prograoner to develop. 
Secondly/ such routines required a substantial conodLtnent of 
time ai^ resouri^^ Were Justified only If used extensively. 
For example, one group claimed to have devoted approximately 1,000 
hour^ to the development of a "driver" which now allows theia to 
add an hour of Instructional time In one hour of work. If only 
one hour of such Instruction Is used. It would have required 1,001 
work hours, but If 1,000 hours of such Instruction are used, each 
will have required only 'two work hours. Since It Is doubtful that 
1,000 hours of one format would be ased, the actual figure should 
fall somewhere In between the extremes. 

As can be seen In the above example, some groups have 
written elaborate support routines which were"^ ln t ended fo r 
general or repeated use» while other groups have used "quicksand 
dirty" mechoda with no generalizability. Several of these efforts 
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wen later mede ob901ete by new system features » such ae the 
system<*provlded router* In the future, support routines should 
require less time and effort because of the availability of 
existing routines 9 but some unique situations » requiring new 
routines or adaptations of existing routines » are likely. Also, 
the application of existing routines to the particular courseware 
could demand the attention of a programming specialist. 

Characteristics of groups » 

In addition to the models of group structure^ groups differed 
In their methods of selecting members of the group and the 
training received by the new n^nbers. In maiqr cases^ these 
characteristics came about as the result of gradual development 
rather than on the basis of overt decisions. 

Selection methods * When a project relied upon subject 
matter specialists of professional status, especially faculty 
meoiberst to volunteer for involvement in courseware production, 
the only selection possible was self-selection. If there were no 
restrictions on the length of time, the computer availability, or 
miober of the support personnel, such a procedure allowed the most 
motivated and capable developers to emerge. However » there was a 
tremendous risk of wasted effort. Some incapable persons 
continued to work because of the glamour of the medium, a greater 
chance for advancement, or a variety of personal reasons. Some 
capable persons discontinued their efforts because of difficulties 
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•tenming^ fron conpetitlon^ for United resources, desires to use 
their time in the most profitable ways, or personal reasons* The 
self-selection approach must account for these failures, as veil 
as the successful developers, ^en determining costs and 
efficiency* 

When the leaders of a courseware development group ivere 
faced with deadlines or liodted resources, as was usually the 
case, they had to make choices among available personnel* Vhsn 
*:he lesson designers were hired on a full-time basis, released 
from teaching or oth^r duties, paid as consultants, 6r even Just 
given access to resources, some type of overt selection also took 
place* Since none of these Jobs were very routine or well defined 
the first criterion was interest or motivation* Relevant 
backgrounds are discussed in a later section, but beyond these 
specific skills and experiences, personal characteristics such 
as flexibility, prolonged dedication to a task, and interest in 
innovation were considered* 

At the time of this study it was necessary to observe each 
person workini; on the PLATO system in order to determine the 
eventual level of productivity. Consequently, many groups had 
to readjust personnel commitments when people who were hired did 
not prove to be productive* One of the people most heavily 
Involved in the training of new lesson designers and programmers 
claimed to be able to predict future progranonlng ability wlth^9QZ 
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aceuraey after th« first day of training but not on tba basis of 
previous y documented background « If such predictions are possible^ 
It should also be possible to devise a task-specific test which 
eould aid in these judgments before training begins (Pophaia, 
et al .. 1974). 

TPIOR training methods # The one area of competency for 

r 

vhich specific training programs were developed was knowledge of 
the TUTOR language « Such training also include PLATO lesson 
design^ which is a pQart of instructional design* During the 
early days of PLATO IV and throughout the lif« of PIATQ III» such 
training was very Informal « Evidently, most people who expressed 
Interest were given a demonstration of some existing lessons^ 
shown how to ''sign on," given a lesson space with which to work, 
and told to ask questions whenever they needed help. Stech people 
went through whichever existing lessons they heard about (as a 
student and, perhaps, looking at the TUTOR code), reviewed the 
scant documentation that existed online, and asked whoever was 
sitting at the next terminal if a problem arose. They soon 
learned who were the most knowledgeable people, including the 
systems programodng staff, when they got to more complex 
questions* This informality was ndt only feasible because of the 
.lose proximity of all PLATO users, but it was also necessary 
because the language was undergoing almost dally revulsion, often 
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In response to the questions of the users* Seversl factors such 
as the Increasing stsblllty of the TUTOR language, more people at 
remote locstlons, and th^ organization of formal courseware 
development teems led to the feasibility and need for more formal 
training methods<» 

Three msjor types of training evolved. First, accredited 
courses Which Included training In the basics of TUTOR, as veil as 
general discussions of computer applications In education and 
instructional development methods or en snalysls o£ HATO as a 
coaq>uter system, were offered by several departments of the 
University of 7 ^Inols, Including extension ccurses lir Chicago • 
The courses used e workbook (Ghesqulere, Davis, and Thoiqison, 
1974) end Its accompanying PLATO lessons* 

A second effort was aimed mostly et people mho accepted 
full-time Jobs to develop PLATO meterlals, especielly in the 
military* Most of these people participated in one-* or two-^week 
intensive workshops et the University of Illinois* During the 
workshops, the people learned the basics of the TDTOR language 
and, if necessery, the rudiments of lesson design and validation* 
The workshops used^some lessons and a manual by Bohn (1973)* 
These basic introductions to TUTOR were estimsted to take 20 - 80 
hours of work, depending upon the person ^s background* People 
with counter progrsnming, instructional development, and 
mathematical beckgrounds tended to take less time although such 
backgrounds were not necessary for eventual success* 
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FlnaiL/, adv«np«d trainlhg In the WIOR langoAge, foUowing 
either of; the twd l^vi^uf alt^^ was much tnore Infornal^ 

and was cXosali^ ri1,aC%d t& th« persoiif a actual work. ^Uhan 
problems aroae^ assistance vas sought iTom various publications^ 
such as Sherwoo<l (IfX*) and E. Avner (1975t, 1975b), froa the onMne 
docunsntatloa available In lesson AIDS and elsewhere » and ,ftoai 
various consultants. GERt set up the PLATO Services Organization 
to provide con|(ultetion either thraug}i persotial cdntact or tiirottgl%. 
the various conaunlcatldn ieatuers of the PlklO system. In addition 
nsny groups found it convenient and^ perhapajt necessary to hire or 
develop a local TUTOR expert who could pro^*lde consultation and 
advanced training. The convrehenslveness of a peraonS knowledge 
of TUTOR varied greatly and depended upon both the person* a 

background and the types of lessons being written. It took 

♦ 

anywhere from a few wpeka to a year before the leaaons being 
written were of sufficient quality to warrant keeping and using 
them. When this training was for an individual who waa Joining 
an exiating groiip* the learning often took place while the 
newcoo^ was^ working produi^tivaty under the direction of a 
competent group member, the amoung of direction usually 
decreased as the degree of the newcomer ^s coo^etency increased. 
When a new group waa being formed, there was more pressure to 
advance beyond the learning stage but also less chance for 
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competent suf^ervlsion of individuals. Several groups » therefore^ 
hired a few highly skilled TUTOR progranmer^ to set up a fuirisal 
tralTilng program. 

Characteristics of Individuals. 



W''.;;uin most gro 3^ there were differences in the backgrounds 
sought tor varieus positxons^ in the interests o^ p^eople as ::hey 
became more involved » in the amount o| work expected, and in the 
rewards given for FLATO work. All of these factors can be 
considered as characteristics of x\ !lviduals. 

Relevant backgrounds . The mpdels of group stmcture 
de^cr^.bed earlier ^call for persons with different backgrounds or, 
conversely, were differentially appropriate when different 
backgrounds were fotnid in the development team. In some cases, 
the model came first; in other cases, the people came first; in 
most cases, one or twp people began the effort, decided upon a 
particular working relationship, and then hired people wij^h 
appropriate backgrounds to fill in the rest of the team. 
Regardless of the particular model or coinbination of models, the 
following areas of expertise (or, at least, responsibility) had 
to be included: suliject matter content, instructional development 
including knowledge of appropriate uses of computer-based systems, 
m.vi TUTOR progrr^ng. In addition, expertise in the production 
of audio/visual materials was necessary when audio messages ox 
slides were used. Iftien the materials were intended fcr general 
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use, someone had to be responsible for monitoring student 
performance, soliciting expert or peer reviews, and 
validation. 

Because the . relatlonsh j p between groups and models of 
group structure Is complex. It Is not possible to give the 
relevant backgrounds of people according to the models. 
Therefore, Table 4 lists the relevant backgrounds according to 
the criteria used in orgir^^'i.ly selecting the groups. The first 
four categories of experience indicate levels of formal 
education, and the last three categories Itidicat^ actual working 
experience prior to initial involvement with the PIATO project. 



appendix), but not all people in all grou; s were inti^rviewed. 
Therefore, while these data are generally indicative of rele mt 
backgrounds, specific nutnbers, perticularly the smaller ones, 
may be slightly inaccurate. 

As indicated in T^ble 4, projects at colleges and 
universities were staffed mostly by people wfth advanced degrees 
in computer-relat;ed fields, educational fields, and a variety of 
other contc;.t areas. Alniost all of the advanced degree holders 
working fc; military projects were at one base^ where materials 
for a para^medical program were being developed. With regard to 
working experience, approximately 61% of the people had prior 
teaching experience* The slightly lower 50% for the university 



All interviewees were asked about 
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groups was due to the involvement of studants working under the 
direction of faculty members. The extremely sjdaU number of 
people with prior experience in developing other instructional 
materials is quite remarkable. It is impossible to say whether 
this phenomenon was advantageous or disadvantageous to PLATO 
courseware development efforts. It can Iso not be determined 
whether the unique characteristics of I 3 courseware development 
procedures w^re the cause or the effect of the lack of curriculum 
development experience. Finally, the fact that 68Z of the developers 
had little or no previous computer experience seems to be in 
keeping with th^ Idea of developing a computer-based educational 
system that can be used by «sany people. However, It Is also 
clear that, even though the system developers are not Included 

o 

In these figures, a significant n «aiber of developers did (and 
perhaps had to) have substantial computer-related backgrounds. 

' Insert Table 4 about here 

V----- — 

Levels of Individual interest . In addition to learning the 
TUTOR language and any other necessary cofflpe tenches, there 
appeared to be several levels of interest through whlrh PLATO 
courseware developers proceeded. In the first level, the person 
learned the basics of TUTOR and the PLATO system. In the second 
level, the interest centered on exploring the TUTOR language and 

i 
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the capabilities of FIATO. There seemed to be a fascination with 
the hardwire igi id sof t ware, and some people, particularly those 
without experience as Instructors, spun-off to becofse TUTOR 
experts or computer system programmers and developers. Others 
without Instructional experience remained this level and worked 
as TUTOR programmers or coders under close supervision. The 
third level was characterized by an Interest In the design of 
individual FIATO lessons in order to make them instructional ly^ 
sound rather than simply interesting things to program* If the 
person was actively teaching at the time, there was often an 
attempt to try various ideas in the classroom* People at this 
level can be called lesson designers. Some people went back and 
forth between the second and third levels as new instructional 
ideas necessitated better programming skills and better 
programming skills generated new instructional Ide^s* A similar 
pattern was found by AvAer (personal communication) in an 
isi4>ubllshed study of 27 first-year PLA'^0 developers* Those 
people who went into the fourth level began to look beyond 
individual lessons and think about sequencing* routing* the 
relationship of lessons to a curriculum* the use of lessons in 
various classroom contexts, and the principles of learning which 
underlie the materials. Finally, the few people who entered the 
fifth level became interested in the design of instructional 
systems. They developed opinions and ideas not only about the 
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design of the instructional materials but also about the process 
of developing such materials and the organization of appropriate 
development teams. Some people with prior experience in 
curriculum development juiq>ed from the first level* to the fourth 
or fifth level » and a few of these later vent back to learn the 
specific skills of the second and, possibly, third levels. 

Full-* time vs# part«time work . The circuowtances under whldh 
a group was formed largely determined whether the meoibers workell 
full-time or part-time. When a newsgroup wa« brought together V 
for the specific purpose of developing courseware for a particular 
project 9 the menbers usually worked full-time* Students who 
worked as programming assistants or support specialists worked 
part-time. The real question arose with regard to the role of 
faculty members. Some groups found that professionals were more 
productive when they were not distracted by other responsibilities 
On the other hand, continued contact with students of the target 
population sometimes stiimilated ideas, tempered them^vith reality, 
and provided a useful forum for trying out new approaches. The 
role of the professional was iiq>ortant« If the person was to be 
only an Initiator and/or reviewer, part-time work was mch mr>re 
fea»)ble and probably preferable than' if ztual production was 
also part of the responsibilities. 
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Vhen a project hid the coiq>lete freedom to detenoine its 
pereoknel composition, the goals of the project detennl.ied the 
deslrAlllty of full-time or part-time workers. When the major 
or sole4oal was the production of appropriate courseware » 
full«*time concentration by professionals » programmers » and 
specialists was needed. When the education of students in th« 
use of computers for instruction was a goal, part-time prograoners 
and specialists could be learning while doing. When coimni tm e n t 
to use of the system by faculty meoibers was a goal, wider 
involvement by part-time professionals was used. A final 
consideration was that when a project was Just beginning, it was 
easier to ter .linate an unproductive part-timer than an unproductive 
full-time employee. 

Rewards for PLATO work . The reward structure of the 
sponsoring institution determined the extrinsic rewards for FLACO 
work. In the military and projects with external funding at 
academic Institutions, there were usually full-time Jobs for the 
duration of the development effort. Academic institutiona also 
released faculty meinbers from other duties in order to develop 
courseware or set up a formal policy to equate such efforts with 
teaching, research, and publications in the determination of 
promotion and tenure. Unfortunately, when the regular procedures, 
usually involving a committee review, have been left Intact, 
courseware development has often not been equated with research 
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07 Other scholarly work. There are also plans to pay royalties 
to the developers of lessons^ based on the amount of use* Of 
course, if the developers were paid for their vork, the royalties 
nay go to\the sponsoring institution. 

Hany o^ the people developing PLATO courseware were Mtivatedt 
in addition to ^e extrinsic rewards described above, by rewards 
which were intrinsic to the work and intangible* Some people 
sinply enjoyed the CM^lenge of progr^nming a computer to carry 
out a specific task; soim, liked the environment and spirit of a 
developmental effort;., some believed that their efforts will have 

V \ 

t \ 

a significant, positive effect on their students or education in 
general* There were even high sch)^l and undergraduate students, 
who worked as prograoming assistants \n order to get lesion 

spaces in which they could design their Wn materials* These 

\ 

intrinsic rewards seemed to drive oiai^ FLAibi; developers to work 

much more than the normal work week* Such unp'iiid overtime work 

\ 

makes it difficult to assess the costs of development* It should 
not be assumed that the intrinsic rewards were sufficient 
motivation for most people* With the possible exception o£ 
uninformed lesson designers providing minimal supervision for 
highly compettot programming assistants, the time investment for 
this work was too great to expect anyone to do a satisfactory Job 
without released time or regular pay* / 
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-Costs of coursevtre development . - 

As Indicated in the appendix to this paper, the interviews 
did include atteo^ts to get estimates of the number of work-hours 
spent and the amount of material developed* Most interviewees, 
% however, were tmable to give anything but gross estimates* The 
estimates that were received ranged from ten to a thousand 
work-hours for single lessons. Some groups had previously 
determined an estimated range by some unexplained means, and all 
group members quoted the same range. Since thes# figures were 
undocumented, there is no way of knowing vhat activities and^ co 
factors werc^ ^included « n / 

Grimes (1975) has recently reported on the "cost of Initial 
development of PLATO, instruction in veterinary medicine." His 
estimated average cost of $828.00 per instructional ;hour» or 
$1.93 ^er student-contact hour-^ does not Include expfnditures for 
"computer usage and salaries of most Instructors for and with . 
whom lessons were developed • • • except those of released-tlme 
personnel or those of Instructors paid by the PLATO Project*" 
He did include the cost of equipment, including PLATO terminals « 
The scope of the coijurseware is indicated as "more than 50 lessons" 
with "approximately 317 instructional hours" using simulations, 
games, problem-solving programs, and interactive tutorials which 
Si a interspersed throughout the four-year veterinary oMdltlne 
curriculum. The entire project, including the start-up period 
and an unspecified validation effort, had lasted four years at 
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th« tiffl* th« report vas written. Grimes concludes with the 
statement that the. PLATO effort at the "University of Illinois 
College of Veiarlnary Medicine has been developmental . The 
expense of continuing the project at this college or initializing 
• similar project at another college should be much less as a 
result of this experience." 

Dare, et al. (1975) of the Aberdeen group reported on the 
costs of development of HLATO courseware for a machinists' course 
and for a course on the construction and interpretation of tests. 
This accounting includes chargies for the on-line and off-line 
time spent by members of the project staff (including 
administrators") for the period from July 1973 to March 1974 
during which approximately 30 instructional hours of courseware 
were developed. However, it does not include charges for the 
•ssistanctt provided by the Military Training Centers group at 
CERL* Costs for terminals, vogmunicatlons, and computer usage 
are presented separately. The average development time per 
instructional hour was reported as 283.6 hours. This development 
time was costed at $8.00 per hour, a figure reported to be the 
actual average hourly Salary of the project staff, giving an 
average development cost of $2,268.80. Dare, et al . did not 
report estimates of thexosts of development per student contact 
^lour nor did they project the number of students to be trained 
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annually with these materials. Assundng 200 students per year 



would use these materials for five years » the average cost of 



courseware development would be $2.27 per student -contact hour* 



SUMHART AND RECOMffiNDATIONS FOR FURTHER RESEARCH 



Since this study was broad and prellmlnar> in nature* It ralsM more 
questions than it answers, tlopefullyt it will provide a starting 
point for further research efforts. In an effort to promote and 
focus such investigations » this final section^itighlights the Intportant 
findings suggests a number of hypotheses and topics in need of 

research » and discusses appropriate methodology^ 

» 

Hypotheses and Topics 

The following summary and suggestions coincide with the 
outline of the main body of this paper. Interested readers are 
referred to appropriate precejding sections for background 




information* 



I. 



Courseware 



A. Stiiges of courseware dcvclo{5ment 



In order to provide a comprehensible description 



of the variety of procedures used ro develop PLATO 



-eoursewflre, five stages (planning^ productiout 



rcview» validation and documcntatiout and 



implementation) were proposed. A description of 



their components and interactions a3 found in the 



courseware development groups under study la 
included. The primary questions concern the 
generallzabllity of this model as a descriptive 
device and Its usefulness In planning future 
efforts. Other research Is needed to discover 
the best order In which planning specifications 
and rev .ews should be made In various situations* 

the personnel who. should b^ responsible for each 

\ 

stage J and the interactive eih^ects of each stage 
on total production time. At this time* ve 
conclude that all stages should be anticipated 
and Included in every courseware development 
effort. 

Types of lesfons 

Several schemes for classifying types of lessons 
are discussed. Once one has b^en accepted an^ 

lessons have been classified* the hypothesis thit 

I 

more complex and interactive lessons require more 
total levelopm^vit time but are more effective for 
certain educational goals should be tested. The, 
differential effects of complex lessons on the 
various stages of courseware development should 
also be investigated. 
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C. Scope of the courseware 

The proper conditions for various levels of the 
acope dimensions (breadth » depth* and degree to 
vhlch Individual lcs«tons are coordinated) need 
to be specified, and the effects of these levels 
on group structures, total development tlBiey and 
the stages of courseware development need to be 
studied. 

D« Content areas 

More Infoinnatlon on the suitability of content 
areas and topics within content areas for 
computer-based education is needed, 

ft 

II « Organization and management 

A. Changing eaq>hases of group activities 

The first question Is whether or not the four 
emphases discussed (exploration, lesson 
production* curriculum Integration* and 
Impl^entatlon) are realistic, and the second 
Is how progress to the final emphases can be 
encouraged, 

B, Models of group structures 

The four proposed models are the independent 
developer* the colleagueship, the lesson designer 
with programming assistants, and the support • 
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staff* Research questions Include the 



appropriateness and productivity of each ^ 



nodel In different situations, their 
advantages and problems, and the effect they 
have on the eventual courseware and the 
stages of courseware deve^rdpment. 



To research this area, a comparison with other 
coiq>uter-ba8ed systems and other instructional 
delivery systems should be made* Within the 
FLATO system, topics for investigation include 
the advantages, disadvantages, and determinants 
of the following topics: system development 
' occurring simultaneously with courseware 
development, the usefulness of peripheral 
devices (slides, audio, and touch panel) , types 
and sources of advice and informationt and the 
usefulness of support routines, 
p. Characteristics of groups 

The personnel selection and training methods 
used by the groups in this study were described. 
Pre-selection measures of courseware development 
abilities should be developed, and their 
reliability and validity should be determined. 




C« Influences of the FU^fO system 
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The hypothesis that volunteers are more (or less) 
productiv.e than ^ull-time employees should be 
Investigated. The factors which affect the ^ 
length of training t\me should be cl&rlfled. 
Characteristics of Individuals 
Relevant educational backgrounds and work 
experiences, as we^l as the levclb of personal 
Interest 9 were described but need to be 
clarified. For each stage of courseware 
development and eaph model of group structure, 
the advantages and disadvantages of full«*time vs. 
part-tlm^ work aud the rewards for courseware ' 
deyelO|^(iieitt work need to be determined, 
^^^eya^tf of. courseware development 
The personnel time and other costs associated 
with all of the above options should be 
determined. Also, the length of '^calendar*' time, 
as opposed to the number of work-liours, for each 
option Is also needed. The products of these 
efforts should be properly measured. Presumably, 
such a measure would Include the number of 
students who use the materials and the ^ength of 
time per studeh^t, but it may al^o Include the 
educational effectiveness of the materials and the 



value of the other forms of instruction being 
replaced. The costs of PLATO courseware 
development should be compared to the 
development costs for other forms of 
Instruction. 

Methodology 

Becaus"" this study attempted to delve Into a relatively new 
area without many guides for substantive topics or methodology, 
the primary method used for collecting data had to be exploratory 
and flexible. Attempts at designing and distributing a 
questionnaire, looking at usage data recorded by the PLATO system, 
and asking people to keep a record of their activities for a week 
were relatively unsuccessful because the dat^ were either very 
difficult to organize and interpret or they were obviously 
Incomplete and distortions of tha real procedures.. Sevi-structured 
Interviews (see appendix) were finally selected because they could 
be used to examine past stages of development, even though the 
accuracy of recall of ietalls Is questionable under such 
conditions, and could be adapted to the Individual Interviewee* s 
areas of knowledge t Another important consideration was that 
people who were unwilling to spend twenty minutes filling out a 
quest lormalre wete willing to spend one or more hours talking 
with e knowledgeable. Interested colleague. Most of the 
interviews were recorded on audio tapes, but the primary tea rds 
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used in writing this report were summaries written by the 
Interviewers, using the form^n the appendix. The form was 
designed after the first group had been interviewed ind was also 
based on a great number of questions, observations, and hypotheses 

which had been generated by many people over thertprevlous two 

/ 

years. ^ 

Many of the hypotheses and topics of the previous section 
could and should be studied Independently, using a wide variety of 

stionnalras, tests, Interviews, on-line measures, and other 
forms of data. However, the complex and interacting nature of 
many Ljplcs indicates the need for a more comprehensive study. 
In an effort to supplement interviews with mora objective data. 
It Is recommended that a mechanism for collecting data on time 
spent in various activities be made available to all PLATO 
courseware development groups and, so far as possible, to other 
cour.. sware development efforts. 



Insert Table 5 about here 



AT possible mechanism would be an interactive lesson or 
tailored questionnaire which asks each developer to indicate on a 
regular, perhaps weekly, basis how much time has been spent in 
each of several activities. Another possibility would be to have 
an independent observer use time-sampling techniques to gather 
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such data. The categories listed in Table 5 Are a starting point 
but could be modified by each project to ^t individual 
circumstances. For instance , some directors may want to list 
each lesson under development instead of tijie categories of 
instructional lessons indicVted. Presumably , some categories » 
such as "audio message!^" or "user recruitment ," would be 
inappropriate for some people or at some times and could be 
eliminated under such circumstances. Equlp^nt and miscellaneous 
costs could be filled in separately. There would also have to be 
accompanying descriptive information about the courseware 
development model^t being used, the selection procedures^ the 
reward structure , the ^cope of the courseware » the number of 
instructional hours , the expected number of students who would 
use the courseware and the other topics raised in this paper. 
After a sufficient number of projects have used this data 
collection system to analyze their own developmental efforts, 
there could be an anonymous data base which cjuld be used by 
new projects to predict the consequences of various decisions* 
Through such a process, courseware development procedures could 
be Improved just as the courseware products of these efforts are 
improved. Hopefully, this paper is a step in that direction. 
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Footnotes 
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Table 1 

Description of the Study Sample 





Groups 


Dates Of 

Intervleiurs 


No. of People 
Intervl ewed 




Elementary 
Reading 


\ 

June, 1974;! 
March, 197? 


8 




Elementary 
tfattlemattcs 


1 

August-September, 
1974 1 


19 




Accounting 


June, 1974; | 
December, t974 


4 


NSF 


Biology 


December, 1974; 
March, 1975 


4 


Project 


Chemistry 


July, 1974; 
December, 1974; 
March, 1975 


5 




Community 
College 
English 


July, 1974; 
December, 1975 


11 




Community 
College 
Mathematics 


July, 1974; 
December^ 1974 


6 


ARPA 

P^roject 


Aberdeen 
Proving 
Grounds 

Chanute Air 
Force Base 


December, 1974 
January, 1975 


6 
9 




Sheppard Air 
Force Base 


February, 1975 j 


11 



\ 
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Table l—Contlnued 





Groups 


Dates of 
Interviews 


No. of People 
Interviewed 




Military 
Training 
Centers 


August, 1974 


9 


Support 
Groups 


FLATO Services 
Organization 

PLATO 
Educ at Icnal 
Evaluation 
and Research 


August, 1974 . 

August, 1974 

* 


* /' 

/ 

4 




Basic Medical 
Sciences 


February <-Mar ch , 
1975 


9 


University 
Groups 


Foreign 
Languages 


July, 1974 


7 




Veterinary 
Medicine 


August, 1974 


6 
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Figure 1 
Sm^fs o£ Courseware Development 
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Revi^' 



77 



Valldatibn 
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objccjt^ves 



Specify 
instructional 
strategy 



Specli^ 
program* 
structure 



.lleflne to a 
complete lesson 



ReyieW by 

other 
developet^ 



Reyiew by 
exteirnax 
experts 



Kevlew by 
Instructor^ 
users 



Initial 
student 
testing 



Lutge-Bcale student testing 



Colltect and analytp data [f 



Wrltlb reports and/or doetiSttMrntatlon 



I^pleatentat^Um 



J "Vikalntalii- 
li)Bf$ona in 
jiorUing oi^der 



ii 



Disseminatb 
l>bsson4 to 
tituire users. 



tfp^Ate lessons 
to reflect ne«r 
knowledge 
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Tfible 2 



Loiis 



Number of Groups Using Varic 
Reviewers and-Tarsets of Revi^^ 



Targets of 
Review 


Reviewers 


Othev 
Developets 


< External 
Experts 


Instructor- 
u iers 


Student 
Testing 


Content 


12 


3 


7 


8 


Instructional 
Design 


12 

< 


3 ' 


• 

5 


11 


Coding 


^0 


1 


0 


7 
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Table 3 

Comparison of Models of Group Structure 
and PLATO Groups 





Models 




Groups 
(With 
Starting 
Dates) 


Independent 
Developer 


Colleagueship 


Leffson 
Designer 
with 
Programming 
Assistants 


Support 
Staff 


Foreign 
languages 
(Sept., 1966) 


Phase 1 


Phase 1 


Phase 1 


Fnase i 


Chemistry 
(Sept., 1967) 




Phase 1 


Phase 1 


— • 


f 


Biology 
..(Septe, 1968) 


Phase 1 


Phase 1 


Phase 1 


-/ 

f 

\ 


Kl Ainant ATV 

School 
Mathematics 
(Sept., 1968) 


Phas^ 1 




Phase 1 






Phase 2 




— 




riiase J 




Phase 3 


Veterinary 
Medicine 
(Sept., 1970) 






Phase 1 


Phase .1 


Elementary 




Phase 1 






School 
Reading 
(April, 1971) 




Phase 2 


•Phase 2 


Phase 2 


Conrounity 
College 
English 
(July, 1971) 


Phase 1 


Phase 1 


Phase 1 




Phase 2 


Phase 2 


^hase 2 


Phase 2 




Phase 3 


Phase 3, 


""^hase 3 
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Table 3— Continued 





Models 

<^ 


Groups 
Starting 


Independent 
Developer 


uoixeaguesnxp 


Lesson 
Designer 

Programming 
Assistants 


Support 
Staff 


Community 
College 
Mathematics 
(July, 1971) 


Phase 1 


Phase 1 






Phase 2 

0 


Phase 2 


Phase 2 




Accounting 


ABA 1 

irnase i 








(Sept., 1971) 




— 


Phase 2 




Military 
Training 
Centers 
(July, 1972) 








Phase I 


';hanute Air 
Force Base 
(August, 1972) 




jrnase x 






... 




Phase 2 


Phase 2 


Aberdeen 

Proving ' 
Grounds 
(March, 1973) 




Phase 1 






Basic Medical 
Sciences 

(jMly, 1973) 






Phase 1 


M MB 




Phase 2 


Phase 2 


Phase 2 


★PLATO 

Services 
Organization 

(August, 1973) 








Phase 1 
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T&ble 3"Contlnued 





Models 


Groups 
(With 
Starting 
Dates) 


Independent 
Developer 


Colleagueshlp 


Lesson 
Designer 
with 
Programming 
Assistants 


Support 
Staff 


**PLATO 
Educational 
Evaluation and 
Research 
(August, 1973) 








Phase 1 


Sheppard Air 
Force Base 
(Feb., 197A) 




Phase 1 




Phase 1 



* The people in this group functioned as consultants as early as 
1971, but a formal organization was noc established until 
August, 1973. 

Successor to the CERL Evaluation Office, which was begun In 
August, 1967. 
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Table 4 

Relevant Backs,round Characteristics 





NSF- 
funded 
Projects 


ARPA- 
funded 
Projects 


CERL 
Support 
Groups 


University 
Groups 


•rc»:als 


High School 
Graduates or 
Students 


7 


8 


0 


0 


15 


College 
Graduates 


13 


10 


1 


5 


29 


Masters or 
Equivalent 
Degrees 


27 


4 


6 


7 


44 


Doctorates 


12 


4 


8 


10 


34 












Teaching 
Experience 


36 


16 


11 


11 


74 


Curriculum 
Development 
Experience 


4 


4 


1 


1 


10 


Non-FLATO 
Computer 
Experience 


21 


5 


7 


6 


39 
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Table 5 

Categories for Collecting Courseware Development Data 





Personnel 
Time 


Equip- 
ment 
Costs 




Director 


Professionals 


Junior Staff 


Support Staff 


Computer Access 


Other Equipment 


Miscellaneous 
(Travel, 
Supplies, etc.) 


START-UP: 

Selecting Developers 
















Training Others and Consulting 
















Being Tranled and Keeping Up 
















Exfilorlng System Carabllltles 
















PLAIjlRlNG; 

pacifying Content 
















Specifying Instructional Strategy 
















Specifying Program Stiructure 
















PRODUCTION: 

Instructional Lessons: 
Drill anid Practice 
















Tutorial 
















Games and Simulations 
















Other 
















Support Lessons: 
Routers 
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Table 5- -Continued 



r 

J 


Personnel 
Time 


Equip - 
. ment 
Costs 




Director 


Professionals 


Junior Staff 


1 Support Staff 


Computer Access 


Other Equipment 


Miscellaneous 
(Travel, 
Supplies, etCe) 


















Drivers 












• 




Data Collection 














• 


Other Media: 

SlidesAlicrofiche 
















Audio Messages 
















Workbooks 
















Other 
















REVIEW: 

Reviewing Lessons by Other 
Developers 
















Feedback From Other Developers 
















Feedback From External Experts 
















Feedback From Instructor-Users 
















Feedback From Initial Student-Users 
















VALIDATION & DOCUMENTATION: 

Study Design — ^ 
















Instrument Development 
















User Recruitment 
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Table 5— •Continued 





Personnel 
Time 


Squip* 
ment 
Costs 




Director 


i 

C9 
(0 

o 


Junlor\ Staff 


Support Staff 


Computer Access 


Other Equipment 


• 

u 

u « 
o 

C « m 
«l ^ « 

^ >r'4 

01 < o. 

0 M Q. 

01 H r3 


Data Collection^ 
















Data Analysis 
















Documentation 
















IMPLQfENTAtlON: 
Maintenance 
















Dissemination 
















Updating 
















HMUWABC/SOFTWARE : 
Development 
















Maintenance 
















ADMINISTRATION/COORDINATION 

















9U 
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Appendix 
Individual Interview Summary 



Name 

Curricultm Group 

Position 

Appointments 
Z time (in each) 
Title on project 
Starting date 

Background 
Education 
Experience 

Previous PLATO work 

Programming 

Teaching 

Curriciilum design 
Initial expectations 

Product 

(ffhat is it (programs » parts of programs, content)? 
Example (if tangible) 
Other products 

Criteria for judging products 
How many instructional hours? 
How many work-*hour8 to produce them? 

Role and Process 

Role in production of courseware 
Degree of autonomy 
Nonproductlon activities 

Interaction vlth instructor-users (demos, workshops, recruiting) 
Interaction with professional community (conferences, reading/writing 
publications 

Administration (personnel, coordination of meetings, reports, external 

evaluation) 
Proctoring of students 
Advising (TUTOR, lesson design) 
Training 

Other (hardware development, etc.) 

Inputs Received 

Kinds and sources 
How are they used? 

Review ^ ^cedures (criteria for acceptance of your work) 

Online /of f line methods and behavior 
Uhat is done online 
What is done offline 



4; 



«5 



Use of Time 

Time devoted to single tasks (continuous? Interrupted?) 
Partition of work load (uniform? peaks?) 

Ideal Production Model 
Division of roles 
Chairacterlstlcs of staff 
Other comments 

± of TUTOR 
Learning time and conditions 
Original 

Continuing . . 

. Level of skill (check one) 

Uses all tools" (common, algorithms » answer processing » data) 
Uses some *'tools** 

Writes structured (sequenced) lessons 
Writes linear lessons 
Typing (some coding) 

No editing (slgn-*on and communications only) 
Correspondence between skill level and Job requirements 
Kind of programming 

Instructional 

Support 

Revision 

Use of resources/work style 
Flowcharts 
Printouts 
AlOS/mantials 
Consultation 

TUTOR as aid or hindrance to Instructional design 

Attitudes 

Motivations and rewards 
Principal dislikes about job 
Long-term Impact of PLATO 

Impact of PLATO on Individual (benefits » side effects) 
Comments 
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